Skip to content

refactor: split oversized files into focused modules (ALIEN-291)#133

Open
lilienblum wants to merge 31 commits into
lilienblum/alien-291-remote-bindingsfrom
refactor/split-large-files
Open

refactor: split oversized files into focused modules (ALIEN-291)#133
lilienblum wants to merge 31 commits into
lilienblum/alien-291-remote-bindingsfrom
refactor/split-large-files

Conversation

@lilienblum

@lilienblum lilienblum commented Jul 19, 2026

Copy link
Copy Markdown
Contributor

ALIEN-291 supporting refactor. This PR is stacked on #156, which owns the remote-storage feature changes.

Summary

Splits oversized Rust and TypeScript source files into focused modules and adds one canonical source-length guard shared by local hooks and CI.

  • Keeps external public declarations and import paths stable.
  • Preserves AWS, Azure, and GCP worker controller handler attributes; thin parent wrappers delegate to same-named implementations because the controller macro requires the annotated methods in the parent module.
  • Preserves the ALIEN-291 remote-storage dependency-cycle fix while placing Terraform dependency application in a dedicated 175-line module.
  • Adds a 1,000-line limit for new source files and exact no-growth caps for pre-existing oversized files.
  • Excludes only documented generated output: client-sdks/** and packages/sdk/src/worker-runtime/generated/**.

The largest changed source file is 987 lines. crates/alien-terraform/src/generator/mod.rs is 833 lines after extracting generator/dependencies.rs.

Main splits

  • AWS EC2 and ELBv2 clients
  • GCP Compute client and integration tests
  • AWS S3 integration tests
  • AWS, Azure, and GCP worker controllers
  • CloudFormation, Terraform, and Helm generators
  • deploy up, heartbeat, instance catalog, build, distribution, and registry-proxy modules

Review evidence

  • Public declaration inventory matches the stacked base across every split family; no new external API was introduced.
  • Every original named function/type remains present after the moves.
  • Worker handler metadata is unchanged, and wrapper/implementation sets match exactly: AWS 50/50, Azure 47/47, GCP 64/64.
  • scripts/check-max-lines.sh passes and is invoked by both Fast/Fork CI and local hooks.
  • rustfmt --check and git diff --check pass.
  • cargo check -p alien-terraform --all-targets passes.
  • Terraform lib tests pass: 15/15.
  • ALIEN-291 dependency regressions pass:
    • CloudFormation cfn-lint: 1/1.
    • Terraform AWS/Azure/GCP fmt + init + validate: 3/3 using Terraform 1.12.2.
  • Final-head Fast CI passes on exact head f64cc8a8c8102e8ebee6f09362ad40e45c781e8b.

Exact review range

  • Stacked base: 69568024fa3dc3a2d82e92897b90c89533f3dd0a
  • PR head: f64cc8a8c8102e8ebee6f09362ad40e45c781e8b

Review move-heavy changes with:

git diff 69568024fa3dc3a2d82e92897b90c89533f3dd0a...f64cc8a8c8102e8ebee6f09362ad40e45c781e8b \
  --color-moved=dimmed-zebra \
  --color-moved-ws=allow-indentation-change

Merge order

  1. Merge feat!: enable remote customer-cloud storage bindings (ALIEN-291) #156.
  2. Retarget this PR to main.
  3. Merge this PR after its final Fast check is green.

Break the 7,000-line worker/gcp.rs into a directory module so each
concern can be read and reviewed in isolation:

- gcp/mod.rs: controller struct, compute-operation tracking, and the
  #[controller] handler impl (kept whole for macro state generation)
- gcp/support.rs: naming helpers, error classifiers, heartbeat
  emission, and GcsNotificationTracker
- gcp/helpers.rs: the plain helper-method impl block
- gcp/tests.rs: the controller test module

Pure code motion: moved items are bumped to pub(super) where needed;
external paths are unchanged via the existing worker::gcp re-exports.
@greptile-apps

greptile-apps Bot commented Jul 19, 2026

Copy link
Copy Markdown

Greptile Summary

This PR splits the monolithic gcp.rs (7,070 lines) into four focused modules under a new gcp/ directory, with no logic, signature, or public-API changes — purely code motion plus the minimum visibility and import wiring to compile.

  • gcp/mod.rs retains the #[controller] struct and handler impl block byte-for-byte; GcsNotificationTracker is re-exported via pub use support::GcsNotificationTracker to preserve its external path.
  • gcp/helpers.rs extracts 14 helper methods bumped to pub(super); gcp/support.rs collects free functions, constants, and support types (all pub(super) except GcsNotificationTracker); gcp/tests.rs lifts the inline mod tests {} block with only the mandatory dedent.
  • The split has been verified with cargo check, cargo nextest run (478/478 pass), and a byte-level diff against the original segments.

Confidence Score: 5/5

Pure code-motion refactor with no logic changes; the proc-macro input in mod.rs is byte-for-byte identical and all 478 tests pass.

Every changed file is either a new module boundary file containing code moved verbatim from the original gcp.rs, or the slimmed-down mod.rs that retains the macro-processed handler block unchanged. The only non-motion deltas are the 14 pub(super) visibility bumps (all internal) and two rustfmt reflows from the dedent and a signature wrap — none affecting behavior. The re-export of GcsNotificationTracker correctly preserves the external API path.

No files require special attention; the split is mechanical and the verification evidence (cargo check, nextest, byte-level diff) is thorough.

Important Files Changed

Filename Overview
crates/alien-infra/src/worker/gcp/mod.rs Root module retaining the #[controller] struct and its handler impl block; declares helpers/support/tests sub-modules and re-exports GcsNotificationTracker via pub use support::GcsNotificationTracker.
crates/alien-infra/src/worker/gcp/helpers.rs Extracted helper impl block with 14 methods bumped to pub(super); imports support items via use super::support::* and conditionally imports GcpWorkerState under test-utils feature.
crates/alien-infra/src/worker/gcp/support.rs Free functions, consts, and GcsNotificationTracker type; all items are pub(super) except GcsNotificationTracker which is pub (re-exported from mod.rs) to preserve the external API path.
crates/alien-infra/src/worker/gcp/tests.rs Test module moved out of the inline mod block with mandatory dedent; imports are adjusted to use super:: paths for support utilities, no logic changes.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A["worker/mod.rs\n(crate re-exports)"]
    B["gcp/mod.rs\n(#[controller] struct\n+ handler impl block\n3,985 lines)"]
    C["gcp/helpers.rs\n(helper impl block\n14 pub(super) methods\n1,422 lines)"]
    D["gcp/support.rs\n(free fns, consts, types\n248 lines)"]
    E["gcp/tests.rs\n(#[cfg(test)]\n1,441 lines)"]

    A --> B
    B -->|"mod helpers;"| C
    B -->|"mod support;\nuse support::*;\npub use support::GcsNotificationTracker"| D
    B -->|"#[cfg(test)] mod tests;"| E
    C -->|"use super::support::*"| D
    E -->|"use super::{ get_cloudrun_service_name, ... }"| D
Loading
%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
flowchart TD
    A["worker/mod.rs\n(crate re-exports)"]
    B["gcp/mod.rs\n(#[controller] struct\n+ handler impl block\n3,985 lines)"]
    C["gcp/helpers.rs\n(helper impl block\n14 pub(super) methods\n1,422 lines)"]
    D["gcp/support.rs\n(free fns, consts, types\n248 lines)"]
    E["gcp/tests.rs\n(#[cfg(test)]\n1,441 lines)"]

    A --> B
    B -->|"mod helpers;"| C
    B -->|"mod support;\nuse support::*;\npub use support::GcsNotificationTracker"| D
    B -->|"#[cfg(test)] mod tests;"| E
    C -->|"use super::support::*"| D
    E -->|"use super::{ get_cloudrun_service_name, ... }"| D
Loading

Reviews (1): Last reviewed commit: "refactor: split gcp worker controller in..." | Re-trigger Greptile

Pure git mv of worker/azure.rs to worker/azure/mod.rs with no content
changes, so rename detection and git log --follow stay intact ahead of
the module carve-out.
Break the 6,500-line worker/azure.rs into a directory module so each
concern can be read and reviewed in isolation:

- azure/mod.rs: controller struct, environment-wake retry tracking, and
  the #[controller] handler impl (kept whole for macro state generation)
- azure/support.rs: naming helpers, wait/delay helpers, error
  classifiers, heartbeat emission, and AzureStorageTriggerInfrastructure
- azure/helpers.rs: the plain helper-method impl block
- azure/tests.rs: the controller test module

Pure code motion: moved items are bumped to pub(super) where needed;
external paths are unchanged via the existing worker::azure re-exports.
@lilienblum lilienblum added refactor Behavior-preserving code reorganization rust labels Jul 19, 2026
@lilienblum lilienblum self-assigned this Jul 19, 2026
@lilienblum
lilienblum requested a review from alongubkin July 19, 2026 06:39
Break the 5,000-line worker/aws.rs into a directory module so each
concern can be read and reviewed in isolation:

- aws/mod.rs: controller struct and the #[controller] handler impl
  (kept whole for macro state generation)
- aws/support.rs: naming helpers, error classifiers, heartbeat
  emission, and the load balancer state types
- aws/helpers.rs: the plain helper-method impl blocks
- aws/tests.rs: the controller test module

Pure code motion: moved items are bumped to pub(super) where needed;
external paths are unchanged via the existing worker::aws re-exports.
Break the 5,700-line gcp/compute.rs into a directory module so each
concern can be read and reviewed in isolation:

- compute/mod.rs: module doc, ComputeServiceConfig, and re-exports
- compute/api.rs: the ComputeApi trait (automock still generates
  MockComputeApi here, re-exported unchanged)
- compute/client.rs: ComputeClient and its ComputeApi impl
- compute/types/: the serde data structures, split along the existing
  section banners (operations, network, load_balancer, instance)
- compute/tests.rs: the PSC serde test module

Pure code motion: no items were renamed or reordered; external paths
are unchanged via glob re-exports from compute/mod.rs. Note in
gcp/AGENTS.md that compute/ is the directory-module variant of the
cloudrun.rs pattern.
Break the 4,800-line lib.rs into concern modules so each area can be
read and reviewed in isolation:

- lib.rs: build_stack orchestration, build_resource, and
  build_target_to_file
- push.rs: push_stack, push targets, OCI tarball selection, and
  image index assembly
- cache.rs: artifact cache keys, source hashing, cargo metadata,
  and cached artifact lookup
- base_images.rs: base image resolution, entrypoint/cmd handling,
  and pull/retry classification
- tests.rs: the unit test module, still gated by #[cfg(test)]

Pure code motion: moved items are bumped to pub(crate); the public
API is unchanged via the root pub use of push_stack.
Break the 4,647-line generator.rs into a generator/ directory so each
concern can be read and reviewed in isolation:

- mod.rs: the options types, the generate_helm_chart and
  render_manager_fetch_values entrypoints, and shared JSON/YAML/name
  helpers
- operator.rs: generate_operator_manifest and the operator document
  builders
- values.rs: ChartAnalysis, values.yaml assembly, and cloud-identity
  mapping
- schema.rs: values.schema.json
- templates.rs: the Helm template (.tpl) bodies
- examples.rs: the per-target values examples and README
- tests.rs: the unit test module, still gated by #[cfg(test)]

Pure code motion: moved helpers are bumped to pub(super); the public
API is unchanged (lib.rs still re-exports generate_operator_manifest and
the other entrypoints). The file drops off the hk.pkl max-lines exclude
now that every module is under the 2000-line cap.
Break the 2,262-line generator.rs into a generator/ directory so each
concern can be read and reviewed in isolation:

- mod.rs: the template constants, the RegistrationMode /
  CloudFormationOptions / CloudFormationTarget types, the
  generate_cloudformation_template entrypoint, YAML serialization,
  stack validation, and the shared resource/output helpers
- parameters.rs: the standard/network/compute parameter blocks, the
  supported-region and custom-domain rules, the standard conditions,
  and the parameter-default helpers
- expressions.rs: the kubernetes/network/domains/management settings
  expressions, the JSON <-> CfExpression bridge, and the CfParameter /
  CfOutput constructors
- tests.rs: the unit test module, still gated by #[cfg(test)]

Pure code motion: moved helpers are bumped to pub(super); the public
API is unchanged (lib.rs still re-exports generate_cloudformation_template
and the other entrypoints). The file drops off the hk.pkl max-lines
exclude now that every module is under the 2000-line cap.
The EC2 client lived in a single 3616-line file. Split it into an
ec2/ module mirroring the existing client layout: api (the Ec2Api
trait), client (Ec2Client and its impls plus the cpu-options unit
tests), and types/ grouped by resource (common, network,
security_group, instance, volume, launch_template).

Pure code motion with import redistribution and module wiring; no
behavior change. Remove the file from the max-lines exclude list now
that every resulting file is under the 2000-line cap.
Break the 3,140-line gcp_compute_client_tests.rs into a directory test
target so each area can be read and reviewed in isolation. The binary
name (gcp_compute_client_tests) and all 17 tests are unchanged:

- main.rs: crate doc, module wiring, and the framework smoke test
- context.rs: the ComputeTestContext fixture and its cleanup/wait helpers
- vpc.rs, load_balancing.rs, disks.rs, instances.rs, ssl_proxy.rs: the
  comprehensive lifecycle tests, one area per module
- errors.rs: the not-found/error-mapping tests plus the operation-status
  test

Pure code motion: no test logic changed. Fixture items used across
modules are bumped from private to pub(crate), per-module imports are
consolidated, the mid-function rcgen import is hoisted to the top of
ssl_proxy.rs, and a mislabeled "Error Handling Tests for New APIs" banner
that sat above the SSL lifecycle test is dropped. The file is removed
from the hk.pkl max-lines exclude list now that every module is under the
2,000-line cap.
Break the 2,265-line aws_s3_client_tests.rs into a directory test target
so each area can be read and reviewed in isolation. The binary name
(aws_s3_client_tests) and all 37 tests are unchanged:

- main.rs: the crate attribute and module wiring
- context.rs: the S3TestContext fixture, its cleanup helpers, and the
  put_test_object helper
- bucket.rs: bucket lifecycle, policy, lifecycle-config, public-access
  block, name validation, and notification-config tests
- object.rs: object put/get/head, listing, delete, and empty-bucket tests
- versioning.rs: bucket-versioning and versioned-object tests
- location.rs: the get_bucket_location tests

Pure code motion: no test logic changed. Fixture items used across
modules are bumped from private to pub(crate) and per-module imports are
consolidated. The file is removed from the hk.pkl max-lines exclude list
now that every module is under the 2,000-line cap.
Break the 4,726-line distribution.rs into a distribution/ directory so
each concern can be read and reviewed in isolation:

- mod.rs: DistributionArtifactCleanup, setup_distribution and
  prepare_distribution, the flow-availability and stack-settings
  helpers, the cross-domain flow orchestrators (run_cloudformation_k8s,
  run_terraform_cloud/k8s, run_onprem_k8s), and stack import/finalize
- cleanup.rs: post-error cleanup and retained-resource teardown
- cloudformation.rs: run_cloudformation_aws and the CloudFormation
  request/output/stack-event helpers
- terraform.rs: the Terraform apply/import flow, tfvars, outputs, and
  handoff-debug helpers
- helm.rs: chart/values rendering, the Kubernetes helm target and
  kubeconfig materialization, and image-rewrite helpers
- permissions.rs: the GCP and Azure management permission probes
- env.rs: cloud env assembly (aws/gcp/azure/terraform)
- exec.rs: the process runners and shell-quote helpers
- tests.rs: the unit test module, still gated by #[cfg(test)]

Pure code motion: moved helpers are bumped to pub(super); the public
API is unchanged (setup_distribution and DistributionArtifactCleanup
are still defined in mod.rs). The file drops off the hk.pkl max-lines
exclude now that every module is under the 2,000-line cap.
Break the 2,700-line heartbeat.rs into a heartbeat/ directory so each
family of provider heartbeat types can be read and reviewed on its own:

- mod.rs: the ResourceHeartbeat envelope, ObservedInventoryBatch /
  ObservedResourceSample, shared health/lifecycle/backend/collection
  enums, event snapshots, raw snippets, and metric types
- storage.rs: StorageHeartbeatData and the S3/Blob/GCS/Local variants
- workload.rs: worker, container, and daemon heartbeat data plus the
  runtime unit and pod status types
- cluster.rs: compute-cluster and kubernetes-cluster heartbeat data with
  capacity/drain/fleet and node status types
- data_services.rs: queue, kv, postgres, and vault families
- platform.rs: service-account, network, and remote-stack-management
  families
- registry_build.rs: artifact-registry, build, and service-activation
  families
- azure_platform.rs: Azure resource-group, storage-account, container
  apps environment, and service-bus-namespace families
- tests.rs: the serde round-trip unit module, still gated by #[cfg(test)]

Pure code motion: submodules are glob re-exported from mod.rs so the
public API at the crate root is unchanged. Drop the file from the hk.pkl
max-lines exclude list now that every module is under the cap.
The generator split dedented the tests module by one level to extract it into
its own file, which unintentionally stripped four spaces from a line inside a
raw-string YAML literal, placing deploymentLabelValue at the document root
instead of under logCollector.scope. Helm 4 schema validation rejects the
root-level key, failing log_collector_enabled_chart_lints_and_templates.

Restore the four-space indentation so the literal is byte-identical to
pre-split. Verified with helm 4.2.3 + kubeconform: alien-helm 15/15 tests pass.
Relocate post-split changes from main into their focused modules and extract test modules that crossed the 2,000-line limit after the merge.
…files

# Conflicts:
#	crates/alien-terraform/src/generator.rs
#	crates/alien-test/src/distribution.rs
@lilienblum
lilienblum changed the base branch from main to lilienblum/alien-291-remote-bindings July 23, 2026 10:49
@lilienblum lilienblum changed the title refactor: split oversized files into focused modules ALIEN-291: refactor oversized files into focused modules Jul 23, 2026
@lilienblum lilienblum changed the title ALIEN-291: refactor oversized files into focused modules refactor: split oversized files into focused modules Jul 23, 2026
…ings' into refactor/split-large-files

# Conflicts:
#	crates/alien-infra/src/worker/azure/mod.rs
# Conflicts:
#	crates/alien-test/src/distribution.rs
…ings' into refactor/split-large-files

# Conflicts:
#	crates/alien-cloudformation/src/generator.rs
#	crates/alien-terraform/src/generator.rs
@lilienblum lilienblum changed the title refactor: split oversized files into focused modules refactor: split oversized files into focused modules (ALIEN-291) Jul 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

refactor Behavior-preserving code reorganization rust

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant